Skip navigation
4740 Views 11 Replies Latest reply: Nov 13, 2010 4:44 PM by ptc-1371556 RSS
jhardy1 Copper 7 posts since
Jun 15, 2010
Currently Being Moderated

Nov 8, 2010 3:11 PM

Import binary file to Mathcad 14 - size limit?

I have created a Mathcad 14 worksheet which reads and processes a large binary data file - specifically, a tile of NASA SRTM terrain elevation data. The worksheet works fine using the generally available 3 arc-second resolution files, which are 2.8 MB each, with over 1,000,000 records in each file. (Each file is a vector containing one integer entry for the elevation data of each point on a 1,201 x 1,201 grid, for 1,442,401 integers in total.)

 

I would like to use the same worksheet to process some 1 arc-second data files, which have approximately 9 times the number of records (3,601 x 3,601 = 12,9607,201 grid points), and 9 times the file size (approx. 25 MB each).

 

When I try to point the "Data Import Wizard" at one of the larger files, it first seems to work fine, but then it loses the data from the imported data file, and the worksheet stops working, as it has no data vector in the main data array. (The array shows only a single entry with a zero value, instead of the expected vector of data entries - see attached screen grab.)

 

I suspect this is an issue with the file size exceeding some limit of Mathcad for handling large binary files, and / or exceeding memory limits.

 

Does anyone know what are the effective limits for handling large binary files in Mathcad 14.0 M020? I couldn't find anything in the documentation.

 

Thanks for any help!
(Note: I have also posted this question on the Eng-Tips forum http://www.eng-tips.com/viewthread.cfm?qid=285290&page=1 - please don't feel obliged to respond to both! )
Attachments:
  • A.Non Diamond 9,898 posts since
    May 11, 2010
    Currently Being Moderated
    Nov 9, 2010 5:20 AM (in response to jhardy1)
    Re: Import binary file to Mathcad 14 - size limit?

    With 13 million points it's likely you have simply run out of memory.

     

    An array requires a contiguous block of memory, so it's possible that if you reboot your computer, then start only Mathcad, you may be able to read in the file.

  • LucMeekes Bronze 90 posts since
    Jun 4, 2007
    Currently Being Moderated
    Nov 9, 2010 1:05 PM (in response to jhardy1)
    Re: Import binary file to Mathcad 14 - size limit?

    You may have hit a (or the) maximum allowed array or matrix size.

    In Mathcad 11 the maximum array index is 16,000,000-1 exactly. That's a program(med) limitation. Mathcad 11 tells you when an array is too large. Don't know for Mcad 14.

     

    Regards,

    Luc

    • A.Non Diamond 9,898 posts since
      May 11, 2010
      Currently Being Moderated
      Nov 9, 2010 1:43 PM (in response to LucMeekes)
      Re: Import binary file to Mathcad 14 - size limit?

      In Mathcad 11 the maximum array index is 16,000,000-1 exactly. That's a program(med) limitation.

      I don't think so. The MC 11 help says:

       

      The effective array size limit depends on the memory available on your  system. Approximately 8 bytes of memory are needed per array element. For most  systems, the array limit is at least 1 million elements. In no system is it  higher than 8 million elements.

       

      Except that is definitely not correct, because I just created an array bigger than 8 million elements. Maybe that piece of the help is a left over from an earlier version. I thought the size was limited only by memory (which is certainly the case in MC 14 and MC15).

      • MikeArmstrong Diamond 3,707 posts since
        Dec 3, 2008
        Currently Being Moderated
        Nov 9, 2010 1:51 PM (in response to A.Non)
        Re: Import binary file to Mathcad 14 - size limit?

        I just tried to create an array with an index of 16,000,000 and it failed, then again my work laptop is painfully slow.

         

        Mike

      • LucMeekes Bronze 90 posts since
        Jun 4, 2007
        Currently Being Moderated
        Nov 9, 2010 2:08 PM (in response to A.Non)
        Re: Import binary file to Mathcad 14 - size limit?

        Richard,

         

        The 8 million limit is most probably a left-over from older versions.

        I've determined the limit in Mathcad 11 to be just 1 less than 16,000,000 back in 2006. if you search the old forum for "array", "size" and my name you should find it.

        It's a good thing if PTC has increased the limit to available memory size in latest versions.

         

        Luc

        • A.Non Diamond 9,898 posts since
          May 11, 2010
          Currently Being Moderated
          Nov 10, 2010 7:53 AM (in response to jhardy1)
          Re: Import binary file to Mathcad 14 - size limit?

          I don't know why you are seeing that hard limit. In MC15 I just created a vector with 40,000,000 elements. I can create a matrix with 40,000,000 elements too (altohugh not at the same time!).

          • A.Non Diamond 9,898 posts since
            May 11, 2010
            Currently Being Moderated
            Nov 10, 2010 9:43 AM (in response to A.Non)
            Re: Import binary file to Mathcad 14 - size limit?

            OK. All is not as simple as it seems.

             

            If I shut down Mathcad and restart it I can create two arrays of 40,000,000 elements each. If I try to increase the size to 50,000,000 elements I get an out of memory error. If I close Mathcad and restart it I can now create two arrays of 50,000,000 elements each!

             

            BUT If I change the size to 500 elements the amount of memory Mathcad is sucking up does not change! If I disable the calculations of the arrays the amount of memory Mathcad is sucking up does not change! If I close the worksheet completely the amount of memory Mathcad is sucking up does not change! If I reload the worksheet and try to create the arrays again I get an out of memory error, so even at this point the memory sucked up by the previus worksheet is not freed!

             

            Now Mathcad versions post 11 are based on the .NET framework, and the .NET framework is designed to handle all memory allocation. There's a very nice description of how it works here:

             

            http://www.c-sharpcorner.com/UploadFile/tkagarwal/MemoryManagementInNet11232005064832AM/MemoryManagementInNet.aspx

             

            If Mathcad were relying on the .NET memory management the observed behavior should not be possible. It would not even be possible to have fragmented memory. Well, it would, but only until you need a bigger chunk of memory, at which point .NET would do a garbage collect and compaction to leave you with a big contiguous block of memory. It might be possible that decreasing the array size or disabling the calculation would not free the memory, because it would not force .NET to do a garbage collection. When a worksheet is closed though Mathcad should force a garbage collection. And if I load the worksheet a second time and try to create the arrays again then .NET should perform a garbage collection anyway, without any prompting from Mathcad, because it needs the memory.

             

            So the only conclusion I can come to is that Mathcad has it's own memory management implemented on top of .NET. Further, that memory management is obviously very poor, and that is why everyone mysteriously runs out of RAM when they are not expecting it. I guess one lesson from the above is that shutting down Mathcad and restarting it may drastically improve things. That should not be necessary though!

            Attachments:
            • A.Non Diamond 9,898 posts since
              May 11, 2010
              Currently Being Moderated
              Nov 10, 2010 10:51 AM (in response to A.Non)
              Re: Import binary file to Mathcad 14 - size limit?

              I guess one other possibility is that .NET is being used to manage the memory, but when a worksheet is closed the worksheet object is not being set to "nothing", and so .NET thinks the worksheet is still being used. It's hard to believe that such poor programming could come out of a major software company though.

  • ptc-1371556 Copper 15 posts since
    Oct 2, 2007
    Currently Being Moderated
    Nov 13, 2010 4:44 PM (in response to jhardy1)
    Re: Import binary file to Mathcad 14 - size limit?

    Hi

     

    This issue of running out of memory has been raised a number of times over the years in the collaboratory.  Often blame is fobbed upon NET but this is not right simply because many other programmes run in Windows without memory issues like in Mathcad.  I think we need to get right to the issue here and not skirt around it all the time. The limitations should be documented in its manuals etc as users are getting a false impression of what can be done.

     

    I have  used Mathcad v11, v13, v14 and v15 and the same issue exists across all these versions.  Mathcad falls over when you use large arrays and when you start processing them - after all just creating a large array and doing nothing with it is a totally pointless exercise and proves nothing.  Mathcad must have an internal structural problem with regard to memory, particularly stressing its internal structure when processing large arrays.  It really needs to fix the issue once and for all.  As time goes on, we do more computation and process ever larger sources of data: this is a non-reversible trend.  Mathcad or any other software provider must keep up with these real life demands.  Mathcad has hit the limit so far as I am concerned and it seems for a number of other users as well.  For PTC / Mathcad trading as an provider of high level technical software, this is a poor situation and undermines their credibilty.  

     

    Do not get me wrong, I really like the Mathcad programme.

     

    However, Mathcad is getting very frustrating for me to use and it is even more frustrating to see Mathcad / PTC do so little software development over the last few years to push Mathcad further along the envelope and make it generally more powerful.  Forum users has raised many useful suggestions on advancing Mathcad, many of which should be fairly straightforward to implement.   I do not know what is in Prime (I am concerned by some of the comments in this forum that Prime has some limitations compared to even Mathcad 15), but I think PTC really need to pull their finger out and get some serious development work done.  I suggest that PTC should make some announcements to its users about what it is doing.    Has PTC properly resourced its programming team on software development? - I suspect not given the limited development output over the last few years.  Perhaps there are also issues on how PTC is running on a business management aspect?   It seems to me Mathcad may be good for a certain level of work, but it will get left behind for larger scale work.

     

    I have gone through my learning curve with Mathcad and written a number of programmes over the last 7 years or so for my work.  I am loath to dump all that and re-learn something else like Maple or Mathematica.  So I am hoping that Mathcad does progress and go to another level.  As soon as possible, please!!!

     

    Regards

     

     

    David

More Like This

  • Retrieving data ...

Bookmarked By (0)

Legend

  • Correct Answers - 3 points
  • Helpful Answers - 1 points