Announcement

Collapse
No announcement yet.

How to process multiple input sources data and generate single output

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to process multiple input sources data and generate single output

    Hi Team,

    I am new to Mirth. I am trying to understand how we can develop a channel that can consume data from 5 different sources and generate single output file.

    The Input file format for all the 5 files is Delimited file and the output is single Flat File (Fixed length) multi level hierarchical data.

    Each of the 5 files has a common field to correlate the records and the output needs to be generated by correlating the 5 input files based on this field.

    Thanks in advance.
    Last edited by arao8; 07-28-2020, 10:55 AM.

  • #2
    Firstly, wrong channel, this one is for the development of mirth itself, not the development of channels using mirth. However a short answer to your question is that it is not trivial without an interim data store (e.g. reading those files to a db and then correlating, in which case SQL joins would make that far easier).
    Mirth 3.8.0 / PostgreSQL 11 / Ubuntu 18.04
    Diridium Technologies, Inc.
    https://diridium.com

    Comment


    • #3
      Thanks for the response.. Is there anyway that processing can happen without writing it down to Database. The each input file sizes could go till 3GB.

      Thanks in advance

      Comment


      • #4
        Not really, since you have to correlate somehow/somewhere. But I may not understand what you mean by "correlate". To me that means you might read file1, then get some pertinent value from file 2 related to that tile 1 row, then based on that value get a value from file 3 and so on, eventually writing a row to disk.

        Maybe post a small snippet of each file and explain your algorithm.
        Mirth 3.8.0 / PostgreSQL 11 / Ubuntu 18.04
        Diridium Technologies, Inc.
        https://diridium.com

        Comment


        • #5
          Okay thanks for the response. In the existing environment, we are using IBM Transformation Extended tool where it can accept multiple input files and generate multiple output files.

          We would like to replace the this transformation logic present in IBM ITX maps by writing the same logic in Mirth.
          In ITX map, we are not actually using any DB adapter while doing this transformation and would like to see if the same can be done in Mirth.

          Thanks in advance

          Comment


          • #6
            In theory it is possible, but like I said you need to provide some samples and the algorithm you are using. For example if the correlating is sequential as you read a file that is one thing, if the expectation is to randomly access a row in each file, that is another thing. I can't say I'd use mirth at all, since reading flat files to DBs with native tools that DB vendor provides is normally far faster than an integration engine can load those same files. As is the export of the reconciled data.
            Mirth 3.8.0 / PostgreSQL 11 / Ubuntu 18.04
            Diridium Technologies, Inc.
            https://diridium.com

            Comment


            • #7
              Even a single 3GB file may kill Mirth, not talking about 5 * 3GB.

              Comment

              Working...
              X