Tags: according, box, memory, programming, python, running, theerror, time

Python running out of memory

On Programmer » Python

10,802 words with 2 Comments; publish: Sun, 04 May 2008 21:56:00 GMT; (20058.59, « »)

I am running into issues with python running out of memory According to the

error box that I get from time to time. So I was wondering if anyone

knowlageable about that type of thing could point me in a direction to start

figuring out whats wrong.

The Situation:

My application () has a wxTreeCtrl of nodes that it reads from an xml

file. Feat node when dubble clicked calls a function to generate the correct

type of window. Now if a user has many nested wxNotbook nodes that each have

several wxTextCtrl nodes in them python will crash and run out of memory

after the user has opened and close the node several times. My first thought

was the window was not actualy being destroyed, but I dubble checked the

code and the window is indeedbeing destroyed and recreated each time. So I

am out of ideas for why opening and closing the window multiple times would

cause the system to crash

Mock Code:

gametree(wxTreeCtrl):

EVT_DL_CLICK(self,on_dbl_click)

def on_dbl_click(self,evt)

call_node_handler(evt)

tabber_handler(wxNotebook):

max = GetChildCount(node)

for m in range(0,max):

panel = call_child_node_handler(node)

self.AddPage(panel)

This is a very simplistic example of whats going on, but even from this you

can see that if I have 20 nested wxNotebook nodes each with multiple text

nodes it can go into a rather deep and long for loop. If you would like I

will post the full function path from the initial event to where it crashes

Console Dump:

File "C:\Python24\Lib\site-packages\wx-2.6-msw-ansi\wx\_controls.py", line

177

7, in __init__

newobj = _controls_.new_TextCtrl(*args, **kwargs)

wx._core.PyAssertionError: C++ assertion "wxAssertFailure" failed in

\\src\m

sw\control.cpp(162): CreateWindowEx("EDIT", flags=52010080, ex=00000200)

failed

This consol dum differs from time to time, but not by much, the only

differance I notice of hand is EDIT is sometimes STATIC in the

CreateWindowEX.

System Specs:

Python 2.4.2

wxPython 2.6.2.1

S: winXP

This issues does not happen on my Gentoo install with the same Python and

wxPython versions, which leads me to belive it is something specific to the

windows install of wx or python thats cauing the error

All Comments

Leave a comment...

  • 2 Comments
    • Dj Gilcrease <digitalxero (AT) gmail (DOT) comwrote:

      I am running into issues with python running out of memory According to the

      error box that I get from time to time. So I was wondering if anyone

      knowlageable about that type of thing could point me in a direction to start

      figuring out whats wrong.

      I doubt your problem stems from your wxPython use. I would wager more

      that it stems from

      The Situation:

      My application () has a wxTreeCtrl of nodes that it reads from an xml

      file. Feat node when dubble clicked calls a function to generate the correct

      type of window. Now if a user has many nested wxNotbook nodes that each have

      several wxTextCtrl nodes in them python will crash and run out of memory

      after the user has opened and close the node several times. My first thought

      was the window was not actualy being destroyed, but I dubble checked the

      code and the window is indeedbeing destroyed and recreated each time. So I

      am out of ideas for why opening and closing the window multiple times would

      cause the system to crash

      Mock Code:

      gametree(wxTreeCtrl):

      EVT_DL_CLICK(self,on_dbl_click)

      def on_dbl_click(self,evt)

      call_node_handler(evt)

      tabber_handler(wxNotebook):

      max = GetChildCount(node)

      for m in range(0,max):

      panel = call_child_node_handler(node)

      self.AddPage(panel)

      Always use xrange unless you actually need a list object. all

      platforms, when you use range(), you are creating that many int objects

      which exist simultaneously. This uses up 16 bytes per int object, and

      can (I believe) only be reused by other ints. Compare the memory usage

      of 'for i in xrange(10000000): pass' with 'for i in range(10000000):

      pass'

      I also noticed the following going on in mplay_client.py, class

      client_base, method recvData (as well as mplay_server.py, class

      mplay_server, method recvData):

      while offset != readSize:

      frag = sock.recv( readSize - offset )

      # See if we've been disconnected

      rs = len( frag )

      if rs <= 0:

      # Loudly raise an exception because we've been disconnected!

      raise IError, "Remote closed the connection!"

      else:

      # Continue to build complete message

      offset += rs

      data += frag

      If readSize is large, and the connection is (relatively) slow, you

      will likely get an out-of-memory error during data transfer. Why?

      Because Python allocates a buffer of sufficient size to store the number

      of bytes asked for from sock.recv, and repeated calls to sock.recv()

      won't necessarily re-use the overallocated blocks. There is also that

      little problem of repeated string concatenation via += also being

      horrible for memory use. linux, these out-of-memory errors can cause

      segfaults (due to its malloc implementation), and on Windows they call

      MemoryError exceptions.

      Instead you should use sock.recv(min(readSize-offset, 4096)), or some

      other reasonable constant (512 to 4096 is generally pretty safe), and

      append the results to a list, with a ''.join(data_lst) at the end.

      For methods such as these, you may want to consider subclassing and

      reusing the common code.

      I also notice that the sendMsg code uses the following:

      def sendMsg( self, sock, msg ):

      """Very simple function that will properly encode and send a message to te

      remote on the specified socket."""

      # Calculate our message length

      length = len( msg )

      # Encode the message length into network byte order

      lp = pack( 'i', socket.htonl( length ) )

      try:

      # Send the encoded length

      sentl = sock.send( lp )

      # Now, send the message the the length was describing

      sentm = sock.send( msg )

      return sentm

      Note that sock.send(data) does not guarantee that all of the data will

      be sent. It only guarantees that it will tell you the amount of

      information that was buffered to be sent. When sending large amounts of

      data, sock.send(data) will be less than len(data). In such cases, it is

      not uncommon to (foolishly) repeatedly slice via data = data[sent:], or

      if one is a bit smarter, data = buffer(data, sent), or even

      snt = 0

      while snt < len(data):

      snt += sock.send(data[snt:snt+4096])

      Remembering to choose that constant wisely once again (512 to 4096 is

      also reasonable here).

      This is a very simplistic example of whats going on, but even from this you

      can see that if I have 20 nested wxNotebook nodes each with multiple text

      nodes it can go into a rather deep and long for loop. If you would like I

      will post the full function path from the initial event to where it crashes.

      Console Dump:

      File "C:\Python24\Lib\site-packages\wx-2.6-msw-ansi\wx\_controls.py", line

      177

      7, in __init__

      newobj = _controls_.new_TextCtrl(*args, **kwargs)

      wx._core.PyAssertionError: C++ assertion "wxAssertFailure" failed in

      \\src\m

      sw\control.cpp(162): CreateWindowEx("EDIT", flags=52010080, ex=00000200)

      failed

      This consol dum differs from time to time, but not by much, the only

      differance I notice of hand is EDIT is sometimes STATIC in the

      CreateWindowEX.

      How long is the app running? Is that the entire dump? Has it been

      talking away to other clients/servers during its execution?

      System Specs:

      Python 2.4.2

      wxPython 2.6.2.1

      S: winXP

      This issues does not happen on my Gentoo install with the same Python and

      wxPython versions, which leads me to belive it is something specific to the

      windows install of wx or python thats cauing the error

      Possible, but I doubt it. I created the following test application

      which creates huge trees and nested notebooks to test out your theory.

      There are 10k nodes in each tree, and it has no problems nesting 23

      levels deep on Windows 2k. Anything deeper and it has issues (likely due

      to the ~240k window instances being created

      There's also the possibility the xml parser being less than memory

      efficientWhich begs the question, how do you know you are getting an

      out-of-memory error? I see no exception of the kind in your traceback

      - Josiah

      #treekiller.py

      import wx

      levels = 23

      class Tree(wx.TreeCtrl):

      def __init__(self, parent):

      wx.TreeCtrlinit__(self, parent, -1,

      style=wx.TR_DEFAULT_STYLE|wx.TR_HAS_BUTTNS|wx.TR_H IDE_RT)

      self.root = root = self.AddRoot("Unseen Root")

      for i in xrange(100):

      ch = self.AppendItem(root, "%02i"%i)

      for j in xrange(100):

      _ = self.AppendItem(ch, "%02i"%j)

      self.Expand(ch)

      class Interface(wx.Panel):

      def __init__(self, parent, level=1):

      wx.Panelinit__(self, parent, -1)

      sizer = wx.BoxSizer(wx.VERTICAL)

      self.nb = wx.Notebook(self, -1)

      self.nb.AddPage(Tree(self.nb), "Tree")

      if level < levels:

      self.nb.AddPage(Interface(self.nb, level+1), "Another Level")

      sizer.Add(self.nb, 1, wx.EXPAND)

      sizer.Layout()

      self.SetSizerAndFit(sizer)

      self.SetAutoLayout(1)

      class MyFrame(wx.Frame):

      def __init__(self):

      wx.Frameinit__(self, None, title="Insanely Nested Trees", size=(625, 600))

      self._ = Interface(self)

      if __name__ == '__main__':

      app = wx.App()

      _ = MyFrame()

      _.Show(1)

      app.MainLoop()

      To unsubscribe, e-mail: wxPython-users-unsubscribe (AT) lists (DOT) wxwidgets.org

      For additional commands, e-mail: wxPython-users-help (AT) lists (DOT) wxwidgets.org

      #1; Sun, 04 May 2008 21:57:00 GMT
    • Josiah Carlson <jcarlson (AT) uci (DOT) eduwrote:

      Dj Gilcrease <digitalxero (AT) gmail (DOT) comwrote:

      I am running into issues with python running out of memory According to the

      error box that I get from time to time. So I was wondering if anyone

      knowlageable about that type of thing could point me in a direction to start

      figuring out whats wrong.

      I doubt your problem stems from your wxPython use. I would wager more

      that it stems from

      , forgot to finish my thought

      it stems from other Python memory utilization, unless you are

      creating hundreds of thousands of controls.

      - Josiah

      To unsubscribe, e-mail: wxPython-users-unsubscribe (AT) lists (DOT) wxwidgets.org

      For additional commands, e-mail: wxPython-users-help (AT) lists (DOT) wxwidgets.org

      #2; Sun, 04 May 2008 21:58:00 GMT