I'd appreciate some input. We have a single data file that has about 1,000,000 rows of data spanning 28 time periods in the columns across each row. The file is too large to import at one time, and so we have created 9 different transformation files to pull subsets of the timeperiods using MVAL. The end user does not want to have to break apart the file and/or schedule 9 separate imports to bring this file in each time. So, I've attempted to create a single SSIS package that runs each of the 9 required component imports, but I'm running into the following error any time that I try to run more than one import at once:
An error occurred while running a packageException of type 'System.OutOfMemoryException' was thrown.
I've tried using a ForEach loop to simply loop through each of the transformation files and run an import for each transformation file it finds. But, I understand that SSIS doesn't release memory until the end of the ForEach loop, which would explain an out of memory issue. I then broke it out and specified 9 distinct Convert task and 9 distinct Dump and Load tasks in the package with an order of Convert 1 --> Dump and Load 1 --> Convert 2 --> Dump and Load 2, etc. This results in the same out of memory issue any time I enable it to run more than 1 convert/dump & load task in the package - they are not running simultaneously, but rather sequentially. I don't understand this. In looking at the logs, it looks like it's creating the temp files for the first import and adding them into BPC, and then creating the temp files for the second import but then fails prior to the second load. This will work for a smaller data file, but shouldn't the memory be released after the task and allow subsequent tasks to complete? Any recommendations on how to address this and automate this load?
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.