No announcement yet.


  • Filter
  • Time
  • Show
Clear All
new posts

  • 837s

    Hey, all. Been doing a ton of searching, but not finding what I'm looking for. I apologize in advance if it's out there and my reading/comprehension skills are just lacking...

    Reading a bunch of 837 files. I could potentially be reading in thousands in one go.

    One channel. My source is a file reader that scans a folder for new files.

    The destination is a channel writer with a javascript transformer.

    The javascript uses foreach (seg in msg.children) to move through every line.

    My problem (well, one of my problems) is that it works great for 1 file - 2 files, 5 files - basically a small amount. When I start getting into a hundred files (and my biggest sample file is probably 10MB) I get stuck with a heap error (out of memory).

    Increasing the size of the heap seems like the ideal solution, but my physical memory isn't gonna change. If I have to read hundreds or even thousands of files, sooner or later I'm going to run into the same out of memory error.

    Thanks for reading. Appreciate any thoughts you've got.


  • #2
    Well, I believe you can solve that my just turning down the level of logging you have on the channel so it doesn't store multiple versions of the message(i.e. just store metadata), but then you wouldn't be able to see the messages in the Dashboard if you needed to review them later.


    • #3
      I don't think you shared enough detail. Can you post a channel? I am not sure why you would have a memory problem if you are processing the files one after another.
      Last edited by pacmano; 04-07-2021, 02:02 PM.
      Diridium Technologies, Inc.