Gaudi::IODataManager Class Reference

#include <src/component/IODataManager.h>

Inheritance diagram for Gaudi::IODataManager:

Collaboration diagram for Gaudi::IODataManager:

Classes
struct	Entry

Public Member Functions
	IODataManager (CSTR nam, ISvcLocator *loc)
	the incident service More...
	~IODataManager () override=default
	Standard destructor. More...
StatusCode	initialize () override
	IService implementation: initialize the service. More...
StatusCode	finalize () override
	IService implementation: finalize the service. More...
StatusCode	connectRead (bool keep_open, Connection *ioDesc) override
	Open data stream in read mode. More...
StatusCode	connectWrite (Connection *con, IoType mode=Connection::CREATE, CSTR doctype="UNKNOWN") override
	Open data stream in write mode. More...
StatusCode	disconnect (Connection *ioDesc) override
	Release data stream. More...
Connection *	connection (const std::string &dsn) const override
	Retrieve known connection. More...
Connections	connections (const IInterface *owner) const override
	Get connection by owner instance (0=ALL) More...
StatusCode	read (Connection *ioDesc, void *const data, size_t len) override
	Read raw byte buffer from input stream. More...
StatusCode	write (Connection *con, const void *data, int len) override
	Write raw byte buffer to output stream. More...
long long int	seek (Connection *ioDesc, long long int where, int origin) override
	Seek on the file described by ioDesc. Arguments as in ::seek() More...
Public Member Functions inherited from extends< BASE, Interfaces >
void *	i_cast (const InterfaceID &tid) const override
	Implementation of IInterface::i_cast. More...
StatusCode	queryInterface (const InterfaceID &ti, void **pp) override
	Implementation of IInterface::queryInterface. More...
std::vector< std::string >	getInterfaceNames () const override
	Implementation of IInterface::getInterfaceNames. More...
	~extends () override=default
	Virtual destructor. More...
void *	i_cast (const InterfaceID &tid) const override
	Implementation of IInterface::i_cast. More...
StatusCode	queryInterface (const InterfaceID &ti, void **pp) override
	Implementation of IInterface::queryInterface. More...
std::vector< std::string >	getInterfaceNames () const override
	Implementation of IInterface::getInterfaceNames. More...
	~extends () override=default
	Virtual destructor. More...
Public Member Functions inherited from extend_interfaces< Interfaces...>
	~extend_interfaces () override=default
	Virtual destructor. More...
	~extend_interfaces () override=default
	Virtual destructor. More...

Protected Types
typedef const std::string &	CSTR
typedef std::map< std::string, Entry * >	ConnectionMap
typedef std::map< std::string, std::string >	FidMap

Protected Member Functions
StatusCode	connectDataIO (int typ, IoType rw, CSTR fn, CSTR technology, bool keep, Connection *con)
StatusCode	reconnect (Entry *e)
StatusCode	error (CSTR msg, bool rethrow)
	Small routine to issue exceptions. More...
StatusCode	establishConnection (Connection *con)

Protected Attributes
std::string	m_catalogSvcName
	Property: Name of the file catalog service. More...
int	m_ageLimit
	Property: Age limit. More...
bool	m_useGFAL
	Property: Flag for auto gfal data access. More...
bool	m_quarantine
	Property: Flag if unaccessible files should be quarantines in job. More...
bool	m_disablePFNWarning
	Property DisablePFNWarning: if set to True will not report when a file is opened by it's physical name. More...
ConnectionMap	m_connectionMap
	Map with I/O descriptors. More...
SmartIF< IFileCatalog >	m_catalog
	Reference to file catalog. More...
FidMap	m_fidMap
	Map of FID to PFN. More...
SmartIF< IIncidentSvc >	m_incSvc

Additional Inherited Members
Public Types inherited from extends< BASE, Interfaces >
using	base_class = extends
	Typedef to this class. More...
using	extend_interfaces_base = extend_interfaces< Interfaces...>
	Typedef to the base of this class. More...
using	base_class = extends
	Typedef to this class. More...
using	extend_interfaces_base = extend_interfaces< Interfaces...>
	Typedef to the base of this class. More...
Public Types inherited from extend_interfaces< Interfaces...>
using	ext_iids = typename Gaudi::interface_list_cat< typename Interfaces::ext_iids...>::type
	take union of the ext_iids of all Interfaces... More...
using	ext_iids = typename Gaudi::interface_list_cat< typename Interfaces::ext_iids...>::type
	take union of the ext_iids of all Interfaces... More...

Detailed Description

Definition at line 29 of file IODataManager.h.

Member Typedef Documentation

typedef std::map<std::string,Entry*> Gaudi::IODataManager::ConnectionMap

protected

Definition at line 41 of file IODataManager.h.

typedef const std::string& Gaudi::IODataManager::CSTR

protected

Definition at line 31 of file IODataManager.h.

typedef std::map<std::string, std::string> Gaudi::IODataManager::FidMap

protected

Definition at line 42 of file IODataManager.h.

Constructor & Destructor Documentation

IODataManager::IODataManager	(	CSTR	nam,
		ISvcLocator *	loc
	)

the incident service

Initializing constructor

Parameters

[in]	nam	Name of the service
[in]	loc	Pointer to the service locator object

Returns

Initialized reference to service object

Definition at line 44 of file IODataManager.cpp.

  : base_class(nam, svcloc)
{
  declareProperty("CatalogType",      m_catalogSvcName="Gaudi::MultiFileCatalog/FileCatalog");
  declareProperty("UseGFAL",          m_useGFAL = true);
  declareProperty("QuarantineFiles",  m_quarantine = true);
  declareProperty("AgeLimit",         m_ageLimit = 2);
  declareProperty("DisablePFNWarning", m_disablePFNWarning = false,
                  "if set to True, we will not report when a file "
                  "is opened by its physical name");
}

Gaudi::IODataManager::~IODataManager ( )

overridedefault

Standard destructor.

Member Function Documentation

StatusCode IODataManager::connectDataIO ( int  typ, IoType  rw, CSTR  fn, CSTR  technology, bool  keep, Connection *  con  )

protected

Definition at line 248 of file IODataManager.cpp.

                                                                                                                     {
  MsgStream log(msgSvc(),name());
  std::string dsn = dataset;
  try  {
    StatusCode sc(StatusCode::SUCCESS,true);
    if ( ::strncasecmp(dsn.c_str(),"FID:",4)==0 )
      dsn = dataset.substr(4), typ = FID;
    else if ( ::strncasecmp(dsn.c_str(),"LFN:",4)==0 )
      dsn = dataset.substr(4), typ = LFN;
    else if ( ::strncasecmp(dsn.c_str(),"PFN:",4)==0 )
      dsn = dataset.substr(4), typ = PFN;
    else if ( typ == UNKNOWN )
      return connectDataIO(PFN, rw, dsn, technology, keep_open, connection);

    if(std::find(s_badFiles.begin(),s_badFiles.end(),dsn) != s_badFiles.end())  {
      m_incSvc->fireIncident(Incident(dsn,IncidentType::FailInputFile));
      return IDataConnection::BAD_DATA_CONNECTION;
    }
    if ( typ == FID )  {
      auto fi = m_connectionMap.find(dsn);
      if ( fi == m_connectionMap.end() )  {
        IFileCatalog::Files files;
        m_catalog->getPFN(dsn,files);
        if ( files.empty() ) {
          if ( !m_useGFAL )   {
            if ( m_quarantine ) s_badFiles.insert(dsn);
            m_incSvc->fireIncident(Incident(dsn,IncidentType::FailInputFile));
            error("connectDataIO> failed to resolve FID:"+dsn,false).ignore();
            return IDataConnection::BAD_DATA_CONNECTION;
          }
          else if ( dsn.length() == 36 && dsn[8]=='-' && dsn[13]=='-' )  {
            std::string gfal_name = "gfal:guid:" + dsn;
            m_fidMap[dsn] = m_fidMap[dataset] = m_fidMap[gfal_name] = dsn;
            sc = connectDataIO(PFN, rw, gfal_name, technology, keep_open, connection);
            if ( sc.isSuccess() ) return sc;
            if ( m_quarantine ) s_badFiles.insert(dsn);
          }
          if ( m_quarantine ) s_badFiles.insert(dsn);
          m_incSvc->fireIncident(Incident(dsn,IncidentType::FailInputFile));
          error("connectDataIO> Failed to resolve FID:"+dsn,false).ignore();
          return IDataConnection::BAD_DATA_CONNECTION;
        }
        // keep track of the current return code before we start iterating over
        // replicas
        auto appmgr = serviceLocator()->as<IProperty>();
        int origReturnCode = Gaudi::getAppReturnCode(appmgr);
    for(auto i=files.cbegin(); i!=files.cend(); ++i)  {
      std::string pfn = i->first;
      if ( i != files.cbegin() )  {
        log << MSG::WARNING << "Attempt to connect dsn:" << dsn
        << " with next entry in data federation:" << pfn << "." << endmsg;
      }
      sc = connectDataIO(PFN, rw, pfn, technology, keep_open, connection);
      if ( !sc.isSuccess() )  {
        if ( m_quarantine ) s_badFiles.insert(pfn);
        m_incSvc->fireIncident(Incident(pfn,IncidentType::FailInputFile));
      }
      else   {
        m_fidMap[dsn] = m_fidMap[dataset] = m_fidMap[pfn] = dsn;
        // we found a working replica, let's reset the return code to the old value
        Gaudi::setAppReturnCode(appmgr, origReturnCode, true).ignore();
        return sc;
      }
    }
    log << MSG::ERROR << "Failed to open dsn:" << dsn
        << " Federated file could not be resolved from "
        << files.size() << " entries." << endmsg;
    return IDataConnection::BAD_DATA_CONNECTION;
      }
      return S_ERROR;
    }
    std::string fid;
    auto j = m_fidMap.find(dsn);
    if ( j == m_fidMap.end() )  {
      IFileCatalog::Files files;
      switch(typ)  {
      case LFN:
        fid = m_catalog->lookupLFN(dsn);
        if ( fid.empty() )  {
          m_incSvc->fireIncident(Incident(dsn,IncidentType::FailInputFile));
          log << MSG::ERROR << "Failed to resolve LFN:" << dsn
              << " Cannot access this dataset." << endmsg;
          return IDataConnection::BAD_DATA_CONNECTION;
        }
        break;
      case PFN:
        fid = m_catalog->lookupPFN(dsn);
        if ( !fid.empty() ) m_catalog->getPFN(fid, files);
        if ( files.empty() )   {
          if ( rw == Connection::CREATE || rw == Connection::RECREATE )  {
            if ( fid.empty() ) fid = m_catalog->createFID();
            m_catalog->registerPFN(fid,dsn,technology);
            log << MSG::INFO << "Referring to dataset " << dsn
                << " by its file ID:" << fid << endmsg;
          }
          else  {
            fid = dsn;
          }
        }
        break;
      }
    }
    else {
      fid = j->second;
    }
    if ( typ == PFN )  {
      // Open PFN
      auto fi = m_connectionMap.find(fid);
      if ( fi == m_connectionMap.end() )  {
        connection->setFID(fid);
        connection->setPFN(dsn);
        auto  e = new Entry(technology, keep_open, rw, connection);
        // Here we open the file!
        if ( !reconnect(e).isSuccess() )   {
          delete e;
          if ( m_quarantine ) s_badFiles.insert(dsn);
          m_incSvc->fireIncident(Incident(dsn,IncidentType::FailInputFile));
          error("connectDataIO> Cannot connect to database: PFN="+dsn+" FID="+fid,false).ignore();
          return IDataConnection::BAD_DATA_CONNECTION;
        }
        fid = connection->fid();
        m_fidMap[dataset] = m_fidMap[dsn] = m_fidMap[fid] = fid;
        if (  !(rw==Connection::CREATE || rw==Connection::RECREATE) )  {
          if ( ! m_disablePFNWarning && strcasecmp(dsn.c_str(),fid.c_str()) == 0 )  {
            log << MSG::ERROR << "Referring to existing dataset " << dsn
                << " by its physical name." << endmsg;
            log << "You may not be able to navigate back to the input file"
                << " -- processing continues" << endmsg;
          }
        }
        m_connectionMap.emplace( fid, e ); // note: only if we disconnect does e get deleted??
        return S_OK;
      }
      // Here we open the file!
      if ( !reconnect((*fi).second).isSuccess() )   {
        if ( m_quarantine ) s_badFiles.insert(dsn);
        m_incSvc->fireIncident(Incident(dsn,IncidentType::FailInputFile));
        error("connectDataIO> Cannot connect to database: PFN="+dsn+" FID="+fid,false).ignore();
        return IDataConnection::BAD_DATA_CONNECTION;
      }
      return S_OK;
    }
    sc = connectDataIO(FID, rw, fid, technology, keep_open, connection);
    if ( !sc.isSuccess() && m_quarantine ) {
      s_badFiles.insert(fid);
    }
    else if ( typ == LFN ) {
      m_fidMap[dataset] = fid;
    }
    return sc;
  }
  catch (std::exception& e)  {
    error(std::string("connectDataIO> Caught exception:")+e.what(), false).ignore();
  }
  catch(...) {
    error(std::string("connectDataIO> Caught unknown exception"), false).ignore();
  }
  m_incSvc->fireIncident(Incident(dsn,IncidentType::FailInputFile));
  error("connectDataIO> The dataset "+dsn+" cannot be opened.",false).ignore();
  s_badFiles.insert(dsn);
  return IDataConnection::BAD_DATA_CONNECTION;
}

IIODataManager::Connection * IODataManager::connection ( const std::string &  dsn ) const

override

Retrieve known connection.

Definition at line 221 of file IODataManager.cpp.

                                                                       {
  auto j = m_fidMap.find(dataset);
  if ( j == m_fidMap.end() )  return nullptr;
  auto i=m_connectionMap.find(j->second);
  return (i != m_connectionMap.end()) ? i->second->connection : nullptr;
}

IODataManager::Connections IODataManager::connections ( const IInterface *  owner ) const

override

Get connection by owner instance (0=ALL)

Definition at line 96 of file IODataManager.cpp.

                                                                                  {
  Connections conns;
  transform_copy_if( std::begin(m_connectionMap), std::end(m_connectionMap),
                     std::back_inserter(conns),
                     [](ConnectionMap::const_reference i ) { return i.second->connection; },
                     [&](const IDataConnection* c) { return !owner || c->owner() == owner; } );
  return conns;
}

StatusCode IODataManager::connectRead ( bool  keep_open, Connection *  ioDesc  )

override

Open data stream in read mode.

Connect data file for writing.

Definition at line 106 of file IODataManager.cpp.

                                                                      {
  if ( !establishConnection(con) )  {
    return connectDataIO(UNKNOWN,Connection::READ,con->name(),"UNKNOWN",keep_open,con);
  }
  std::string dsn = con ? con->name() : std::string("Unknown");
  return error("Failed to connect to data:"+dsn,false);
}

StatusCode IODataManager::connectWrite ( Connection *  con, IoType  mode = Connection::CREATE, CSTR  doctype = "UNKNOWN"  )

override

Open data stream in write mode.

Connect data file for reading.

Definition at line 115 of file IODataManager.cpp.

                                                                                {
  if ( !establishConnection(con) )  {
    return connectDataIO(UNKNOWN,mode,con->name(),doctype,true,con);
  }
  std::string dsn = con ? con->name() : std::string("Unknown");
  return error("Failed to connect to data:"+dsn,false);
}

StatusCode IODataManager::disconnect ( Connection *  ioDesc )

override

Release data stream.

Definition at line 138 of file IODataManager.cpp.

                                                       {
  if ( con )  {
    std::string dataset = con->name();
    std::string dsn = dataset;
    StatusCode sc = con->disconnect();
    if ( ::strncasecmp(dsn.c_str(),"FID:",4)==0 )
      dsn = dataset.substr(4);
    else if ( ::strncasecmp(dsn.c_str(),"LFN:",4)==0 )
      dsn = dataset.substr(4);
    else if ( ::strncasecmp(dsn.c_str(),"PFN:",4)==0 )
      dsn = dataset.substr(4);

    auto j = m_fidMap.find(dataset);
    if ( j != m_fidMap.end() )  {
      std::string fid = j->second;
      std::string gfal_name = "gfal:guid:" + fid;
      auto i=m_connectionMap.find(fid);
      m_fidMap.erase(j);
      if ( (j=m_fidMap.find(fid)) != m_fidMap.end() )
        m_fidMap.erase(j);
      if ( (j=m_fidMap.find(gfal_name)) != m_fidMap.end() )
        m_fidMap.erase(j);
      if ( i != m_connectionMap.end() && i->second ) {
        IDataConnection* c = i->second->connection;
        if ( (j=m_fidMap.find(c->pfn())) != m_fidMap.end() )
          m_fidMap.erase(j);
        if ( c->isConnected() )  {
          MsgStream log(msgSvc(),name());
          c->disconnect();
          log << MSG::INFO << "Disconnect from dataset " << dsn
              << " [" << fid << "]" << endmsg;
        }
        delete i->second;
        m_connectionMap.erase(i);
      }
    }
    return sc;
  }
  return S_ERROR;
}

StatusCode IODataManager::error ( CSTR  msg, bool  rethrow  )

protected

Small routine to issue exceptions.

Definition at line 88 of file IODataManager.cpp.

                                                       {
  MsgStream log(msgSvc(),name());
  log << MSG::ERROR << "Error: " << msg << endmsg;
  if ( rethrow )  System::breakExecution();
  return S_ERROR;
}

StatusCode IODataManager::establishConnection ( Connection *  con )

protected

Definition at line 228 of file IODataManager.cpp.

                                                              {
  if ( !con )  return error("Severe logic bug: No connection object avalible.",true);
  
  if ( con->isConnected() )  {
    con->resetAge();
    return S_OK;
  }
  auto i=m_connectionMap.find(con->name());
  if ( i != m_connectionMap.end() )  {
    Connection* c = i->second->connection;
    if ( c != con )  {
      m_incSvc->fireIncident(Incident(con->name(),IncidentType::FailInputFile));
      return error("Severe logic bug: Twice identical connection object for DSN:"+con->name(),true);
    }
    if ( reconnect(i->second).isSuccess() ) return S_OK;
  }
  return S_ERROR;
}

StatusCode IODataManager::finalize ( )

override

IService implementation: finalize the service.

Definition at line 82 of file IODataManager.cpp.

                                    {
  m_catalog = nullptr; // release
  return Service::finalize();
}

StatusCode IODataManager::initialize ( )

override

IService implementation: initialize the service.

IService implementation: Db event selector override.

Definition at line 57 of file IODataManager.cpp.

                                      {
  // Initialize base class
  StatusCode status = Service::initialize();
  MsgStream log(msgSvc(), name());
  if ( !status.isSuccess() )    {
    log << MSG::ERROR << "Error initializing base class Service!" << endmsg;
    return status;
  }
  // Retrieve conversion service handling event iteration
  m_catalog = serviceLocator()->service(m_catalogSvcName);
  if( !m_catalog ) {
    log << MSG::ERROR
        << "Unable to localize interface IFileCatalog from service:"
        << m_catalogSvcName << endmsg;
    return StatusCode::FAILURE;
  }
  m_incSvc = serviceLocator()->service("IncidentSvc");
  if( !m_incSvc ) {
    log << MSG::ERROR << "Error initializing IncidentSvc Service!" << endmsg;
    return status;
  }
  return status;
}

StatusCode IODataManager::read ( Connection *  ioDesc, void *const  data, size_t  len  )

override

Read raw byte buffer from input stream.

Definition at line 124 of file IODataManager.cpp.

                                                                             {
  return establishConnection(con).isSuccess() ? con->read(data,len) : S_ERROR;
}

StatusCode IODataManager::reconnect ( Entry *  e )

protected

Definition at line 179 of file IODataManager.cpp.

                                             {
  StatusCode sc = S_ERROR;
  if ( e && e->connection )  {
    switch(e->ioType)  {
    case Connection::READ:
      sc = e->connection->connectRead();
      break;
    case Connection::UPDATE:
    case Connection::CREATE:
    case Connection::RECREATE:
      sc = e->connection->connectWrite(e->ioType);
      break;
    default:
      return S_ERROR;
    }
    if ( sc.isSuccess() && e->ioType == Connection::READ )  {
      std::vector<Entry*> to_retire;
      e->connection->resetAge();
      transform_copy_if( std::begin(m_connectionMap), std::end(m_connectionMap),
                         std::back_inserter(to_retire),
                         select2nd,
                         [&](Entry* i) { 
                            IDataConnection* c = i->connection;
                            return e->connection!=c && c->isConnected() && 
                                   !i->keepOpen && c->ageFile() > m_ageLimit;
                         });
      if ( !to_retire.empty() )  {
        MsgStream log(msgSvc(),name());
        std::for_each( std::begin(to_retire), std::end(to_retire),
                       [&](Entry* j) {
                            IDataConnection* c = j->connection;
                            c->disconnect();
                            log << MSG::INFO << "Disconnect from dataset " << c->pfn()
                                << " [" << c->fid() << "]" << endmsg;
                       } );
      }
    }
  }
  return sc;
}

long long int IODataManager::seek ( Connection *  ioDesc, long long int  where, int  origin  )

override

Seek on the file described by ioDesc. Arguments as in ::seek()

Definition at line 134 of file IODataManager.cpp.

                                                                                   {
  return establishConnection(con).isSuccess() ? con->seek(where,origin) : -1;
}

StatusCode IODataManager::write ( Connection *  con, const void *  data, int  len  )

override

Write raw byte buffer to output stream.

Definition at line 129 of file IODataManager.cpp.

                                                                           {
  return establishConnection(con).isSuccess() ? con->write(data,len) : S_ERROR;
}

Member Data Documentation

int Gaudi::IODataManager::m_ageLimit

protected

Property: Age limit.

Definition at line 47 of file IODataManager.h.

SmartIF<IFileCatalog> Gaudi::IODataManager::m_catalog

protected

Reference to file catalog.

Definition at line 59 of file IODataManager.h.

std::string Gaudi::IODataManager::m_catalogSvcName

protected

Property: Name of the file catalog service.

Definition at line 45 of file IODataManager.h.

ConnectionMap Gaudi::IODataManager::m_connectionMap

protected

Map with I/O descriptors.

Definition at line 57 of file IODataManager.h.

bool Gaudi::IODataManager::m_disablePFNWarning

protected

Property DisablePFNWarning: if set to True will not report when a file is opened by it's physical name.

Definition at line 54 of file IODataManager.h.

FidMap Gaudi::IODataManager::m_fidMap

protected

Map of FID to PFN.

Definition at line 61 of file IODataManager.h.

SmartIF<IIncidentSvc> Gaudi::IODataManager::m_incSvc

protected

Definition at line 67 of file IODataManager.h.

bool Gaudi::IODataManager::m_quarantine

protected

Property: Flag if unaccessible files should be quarantines in job.

Definition at line 51 of file IODataManager.h.

bool Gaudi::IODataManager::m_useGFAL

protected

Property: Flag for auto gfal data access.

Definition at line 49 of file IODataManager.h.

---

The documentation for this class was generated from the following files:

• GaudiUtils/src/component/IODataManager.h
• GaudiUtils/src/component/IODataManager.cpp