Inheritance diagram for code::DataDirectoryManager:
The manager does not assume that files are complete until a new file is placed in that directory, thus it can deal with files that are being continuously written to.
A major requirement is that file names are lexically increasing over time. Older files will NOT be read. The efficiency of this class increases if you allow it to remove files that have been processed.
(was: w2algs/ldm2netcdf/ LDMAccessor )
Public Member Functions | |
| DataDirectoryManager (const std::string &inputDirectory, bool readAllRawFiles, bool shouldDeleteAfterRead, const std::string &pattern=std::string(), bool search_sub_dirs=false, bool only_complete_files_should_be_handled=false, SmartPtr< ConventionTest > convention_test=new RawConventionTest()) | |
| Constructs an object that will provide access to streaming data via the readData() method. | |
| virtual | ~DataDirectoryManager () |
| void | setAcceptablePattern (const std::string &pattern_in) |
| By setting a pattern, you can ask the accessor to avoid using the strict LDM raw file format ( e.g: 20020131220951.raw) and accept all files that start with this pattern. | |
| void | setMaxPendingFiles (int new_value) |
| How many files can be pending a read? By default, this is INT_MAX in archive mode and 3 in real-time mode. | |
| virtual int | readData () |
| Reads data from the LDM and returns number of bytes read as a result of this call. | |
| virtual void | addDataHandler (SmartPtr< DataHandler > h) |
| virtual void | addDataHandler (SmartPtr< DataHandler > h) |
| Attach a handler for the data that is read. | |
Protected Types | |
| typedef std::map< std::string, SmartPtr< DataDirectoryManager > > | SubDirs |
Protected Member Functions | |
| virtual int | readMoreDataFromFile (const std::string &fileName) |
| returns number of bytes read from the file in this pass. | |
| virtual int | readAndRemoveFile (const std::string &fileName) |
| returns number of bytes read from the file. | |
| virtual int | findFilesToRead (bool readAllRawFiles=true) |
| Adds any subdirectories found to subdir; these will be processed on next readData(). | |
| bool | shouldReadFile (const std::string &fileName) const |
| virtual bool | isFileComplete (const std::string &filename) |
| bool | matchesPattern (const std::string &pattern, const std::string &fileName) const |
| Does the file name match the pattern? | |
| void | addSubDir (const std::string &dirname, bool readall) |
| Start to parse this subdirectory also. | |
Protected Attributes | |
| std::string | inputDir |
| bool | shouldDeleteAfterRead |
| std::string | pattern |
| int | maxPendingFiles |
| bool | searchSubdirsFlag |
| bool | notStreamingData |
| std::set< std::string > | filesToRead |
| std::string | lastFinishedFile |
| SubDirs | subdirs |
| std::map< std::string, size_t > | file_size |
| std::string | lastStartedFile |
| SmartPtr< ProgressiveReader > | reader |
| SmartPtr< ConventionTest > | conventionTest |
| Buffer | dataBuf |
| std::vector< SmartPtr< DataHandler > > | dataHandlers |
Classes | |
| struct | ConventionTest |
| Interface which tests a filename to see if it matches a convention. More... | |
| struct | RawConventionTest |
| ConventionTest object which returns true if fileName is 18 characters long and ends in ".raw". More... | |
typedef std::map< std::string, SmartPtr<DataDirectoryManager> > code::DataDirectoryManager::SubDirs [protected] |
| code::DataDirectoryManager::DataDirectoryManager | ( | const std::string & | inputDirectory, | |
| bool | readAllRawFiles, | |||
| bool | shouldDeleteAfterRead, | |||
| const std::string & | pattern = std::string(), |
|||
| bool | search_sub_dirs = false, |
|||
| bool | only_complete_files_should_be_handled = false, |
|||
| SmartPtr< ConventionTest > | convention_test = new RawConventionTest() | |||
| ) |
Constructs an object that will provide access to streaming data via the readData() method.
If complete_file_only is set, then the attached DataHandlers' processData() method will NOT be called. Instead, handlers will need to read the data themselves on a processFileFinish() command.
| virtual code::DataDirectoryManager::~DataDirectoryManager | ( | ) | [inline, virtual] |
| virtual void code::DataProvider::addDataHandler | ( | SmartPtr< DataHandler > | h | ) | [inline, virtual, inherited] |
Attach a handler for the data that is read.
| virtual void code::DataDirectoryManager::addDataHandler | ( | SmartPtr< DataHandler > | h | ) | [virtual] |
| void code::DataDirectoryManager::addSubDir | ( | const std::string & | dirname, | |
| bool | readall | |||
| ) | [protected] |
Start to parse this subdirectory also.
dirname is relative to this directory.
| virtual int code::DataDirectoryManager::findFilesToRead | ( | bool | readAllRawFiles = true |
) | [protected, virtual] |
Adds any subdirectories found to subdir; these will be processed on next readData().
Reimplemented in code::DataStreamManager.
| virtual bool code::DataDirectoryManager::isFileComplete | ( | const std::string & | filename | ) | [protected, virtual] |
| bool code::DataDirectoryManager::matchesPattern | ( | const std::string & | pattern, | |
| const std::string & | fileName | |||
| ) | const [protected] |
Does the file name match the pattern?
| virtual int code::DataDirectoryManager::readAndRemoveFile | ( | const std::string & | fileName | ) | [protected, virtual] |
returns number of bytes read from the file.
| virtual int code::DataDirectoryManager::readData | ( | ) | [virtual] |
Reads data from the LDM and returns number of bytes read as a result of this call.
Typically, you will keep calling readData() indefinitely in real-time, and until it returns zero for archive.
Implements code::DataProvider.
Reimplemented in code::DataStreamManager.
| virtual int code::DataDirectoryManager::readMoreDataFromFile | ( | const std::string & | fileName | ) | [protected, virtual] |
returns number of bytes read from the file in this pass.
| void code::DataDirectoryManager::setAcceptablePattern | ( | const std::string & | pattern_in | ) |
By setting a pattern, you can ask the accessor to avoid using the strict LDM raw file format ( e.g: 20020131220951.raw) and accept all files that start with this pattern.
| void code::DataDirectoryManager::setMaxPendingFiles | ( | int | new_value | ) |
How many files can be pending a read? By default, this is INT_MAX in archive mode and 3 in real-time mode.
It has to be a number greater than 1 i.e. at least one file should be allowed to be pending.
| bool code::DataDirectoryManager::shouldReadFile | ( | const std::string & | fileName | ) | const [protected] |
Buffer code::DataDirectoryManager::dataBuf [protected] |
std::vector< SmartPtr<DataHandler> > code::DataProvider::dataHandlers [protected, inherited] |
std::map<std::string, size_t> code::DataDirectoryManager::file_size [protected] |
std::set< std::string > code::DataDirectoryManager::filesToRead [protected] |
std::string code::DataDirectoryManager::inputDir [protected] |
std::string code::DataDirectoryManager::lastFinishedFile [protected] |
std::string code::DataDirectoryManager::lastStartedFile [protected] |
int code::DataDirectoryManager::maxPendingFiles [protected] |
bool code::DataDirectoryManager::notStreamingData [protected] |
std::string code::DataDirectoryManager::pattern [protected] |
SmartPtr<ProgressiveReader> code::DataDirectoryManager::reader [protected] |
bool code::DataDirectoryManager::searchSubdirsFlag [protected] |
bool code::DataDirectoryManager::shouldDeleteAfterRead [protected] |
SubDirs code::DataDirectoryManager::subdirs [protected] |
1.4.7