How to debug segfaults in Ubuntu

Actually, turns out it's not that hard!

You need to do the following:

1. Firstly, create a build folder.
   

   chris@ubuntu:~/src$ mkdir sqlitebuild

   
   
2. Now download the source package:
   

   chris@ubuntu:~/src$ cd sqlitebuild
   chris@ubuntu:~/src/sqlitebuild$ sudo apt-get source sqlite3



3. Now you need to build any package dependencies:
   

   chris@ubuntu:~/src/sqlitebuild$ sudo apt-get build-dep sqlite3

   
   
4. Now extract the source files from the dsc file:
   

   chris@ubuntu:~/src/sqlitebuild$ dpkg-source -x foo_version-revision.dsc

   
   
5. We don't want to strip the debug symbols, we don't really need to worry about build tests, and we don't want optimizations (they get in the way of the backtrace), so we need to set theDEB_BUILD_OPTIONS:
   

   chris@ubuntu:~/src/sqlitebuild$ cd sqlite3-3.6.22
   chris@ubuntu:~/src/sqlitebuild/sqlite3-3.6.22$ sudo /
   > DEB_BUILD_OPTIONS="nocheck noopt nostrip" /
   > fakeroot debian/rules binary
   
   sudo DEB_BUILD_OPTIONS="nocheck noopt nostrip" fakeroot debian/rules binary
   

6. It will now spit out lots of compilation info.

Now you need to actually install the package, which can be tricky if you've already got it installed.

1. Most Ubuntu/Debian packages have dependencies. You'll need to force the uninstallation of the package:
   

   chris@ubuntu:~/src/sqlitebuild/sqlite3-3.6.22$ cd ..
   chris@ubuntu:~/src/sqlitebuild$ sudo dpkg --purge --force-depends sqlite3
   [sudo] password for chris:
   (Reading database ... 333636 files and directories currently installed.)
   Removing sqlite3 ...
   Processing triggers for man-db ...
   chris@ubuntu:~/src/sqlitebuild$ sudo dpkg --purge --force-depends libsqlite3
   dpkg: warning: ignoring request to remove libsqlite3 which isn't installed.
   chris@ubuntu:~/src/sqlitebuild$ sudo dpkg --purge --force-depends libsqlite3-0
   dpkg: libsqlite3-0: dependency problems, but removing anyway as you requested:
   libsqlite3-0-dbg depends on libsqlite3-0 (= 3.6.22-1).
   libmono-sqlite2.0-cil depends on libsqlite3-0 (>= 3.6.13).
   libsqlite3-dev depends on libsqlite3-0 (= 3.6.22-1).
   evolution-indicator depends on libsqlite3-0 (>= 3.6.22); however:
   Package libsqlite3-0 is to be removed.
   libmono-sqlite1.0-cil depends on libsqlite3-0 (>= 3.6.13).
   evolution-couchdb depends on libsqlite3-0 (>= 3.6.22).
   libpackagekit-glib2-12 depends on libsqlite3-0 (>= 3.6.22); however:
   Package libsqlite3-0 is to be removed.
   evolution depends on libsqlite3-0 (>= 3.6.22); however:
   Package libsqlite3-0 is to be removed.
   libedata-book1.2-2 depends on libsqlite3-0 (>= 3.6.22).
   libwebkit-1.0-2 depends on libsqlite3-0 (>= 3.6.22).
   evolution-plugins depends on libsqlite3-0 (>= 3.6.22); however:
   Package libsqlite3-0 is to be removed.
   evolution-exchange depends on libsqlite3-0 (>= 3.6.22).
   libcamel1.2-14 depends on libsqlite3-0 (>= 3.6.22).
   libnss3-1d depends on libsqlite3-0 (>= 3.6.22).
   libedataserverui1.2-8 depends on libsqlite3-0 (>= 3.6.22).
   libopensync0 depends on libsqlite3-0 (>= 3.6.16).
   python2.6-dbg depends on libsqlite3-0 (>= 3.6.22).
   libqt4-webkit depends on libsqlite3-0 (>= 3.6.22).
   python2.6 depends on libsqlite3-0 (>= 3.6.22).
   libsvn1 depends on libsqlite3-0 (>= 3.6.16).
   packagekit depends on libsqlite3-0 (>= 3.6.22); however:
   Package libsqlite3-0 is to be removed.
   evolution-data-server depends on libsqlite3-0 (>= 3.6.22).
   libebook1.2-9 depends on libsqlite3-0 (>= 3.6.22).
   bibledit depends on libsqlite3-0 (>= 3.6.16).
   xulrunner-1.9.2 depends on libsqlite3-0 (>= 3.6.22).
   libaprutil1-dbd-sqlite3 depends on libsqlite3-0 (>= 3.6.22).
   telepathy-gabble depends on libsqlite3-0 (>= 3.6.22).
   libgpod4 depends on libsqlite3-0 (>= 3.6.22).
   libqt4-sql-sqlite depends on libsqlite3-0 (>= 3.6.22).
   libgda-4.0-4 depends on libsqlite3-0 (>= 3.6.22).
   libsoup-gnome2.4-1 depends on libsqlite3-0 (>= 3.6.22).
   banshee depends on libsqlite3-0 (>= 3.6.22).
   python-gpod depends on libsqlite3-0 (>= 3.6.22).
   opensyncutils depends on libsqlite3-0 (>= 3.6.16).
   (Reading database ... 333630 files and directories currently installed.)
   Removing libsqlite3-0 ...
   Purging configuration files for libsqlite3-0 ...
   Processing triggers for libc-bin ...
   ldconfig deferred processing now taking place
   /sbin/ldconfig.real: /usr/lib/debug/usr/lib/libpython2.6.so.1.0-gdb.py is not an ELF file - it has the wrong magic bytes at the start.

2. Next, you install the newly built debs:
   

   chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i libsqlite3-0_3.6.22-1_i386.deb
   Selecting previously deselected package libsqlite3-0.
   (Reading database ... 333625 files and directories currently installed.)
   Unpacking libsqlite3-0 (from libsqlite3-0_3.6.22-1_i386.deb) ...
   Setting up libsqlite3-0 (3.6.22-1) ...
   
   Processing triggers for libc-bin ...
   ldconfig deferred processing now taking place
   /sbin/ldconfig.real: /usr/lib/debug/usr/lib/libpython2.6.so.1.0-gdb.py is not an ELF file - it has the wrong magic bytes at the start.
   
   chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i libsqlite3-0-dbg_3.6.22-1_i386.deb
   (Reading database ... 333632 files and directories currently installed.)
   Preparing to replace libsqlite3-0-dbg 3.6.22-1 (using libsqlite3-0-dbg_3.6.22-1_i386.deb) ...
   Unpacking replacement libsqlite3-0-dbg ...
   Setting up libsqlite3-0-dbg (3.6.22-1) ...
   chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i sqlite3_3.6.22-1_i386.deb
   Selecting previously deselected package sqlite3.
   (Reading database ... 333627 files and directories currently installed.)
   Unpacking sqlite3 (from sqlite3_3.6.22-1_i386.deb) ...
   Setting up sqlite3 (3.6.22-1) ...
   Processing triggers for man-db ...
   chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i sqlite3-doc_3.6.22-1_all.deb
   (Reading database ... 333633 files and directories currently installed.)
   Preparing to replace sqlite3-doc 3.6.22-1 (using sqlite3-doc_3.6.22-1_all.deb) ...
   Unpacking replacement sqlite3-doc ...
   Setting up sqlite3-doc (3.6.22-1) ...
   

Now to debug the segmentation fault. I'm currently looking into why Firefox is segfaulting on me every one and half minutes or so, and I've worked out that because I have a corrupted places.sqlite (index has gone bad) that it's actually sqlite3 that is having the problem.

I've actually applied this same procedure to Firefox, and I determined that sqlite wasn't happy with the following query:

select h.url, v.visit_date, h.hidden, 0 AS whole_entry FROM moz_places h JOIN moz_historyvisits v ON h.id = v.place_id where v.visit_date < '2000-01-01';

From here, it's pretty easy to hook up gdb to get the full backtrace, or even step through the code.

Update: My efforts have resulted in a bugfix from the sqlite team - yay! Basically, the issue was that Firefox was segfaulting every minute and a half or so. It turns out that my places.sqlite datafile was corrupted, the sqlite guys have now developed a fix.

If anyone else has a similar issue to this one, incidentally, then they can't go wrong with the following:

chris@ubuntu:~/.mozilla/firefox/1u64q3v3.default$ for i in *.sqlite; do echo "Reindexing $i"; echo "reindex;" | sqlite3 $i; done
Reindexing content-prefs.sqlite
Reindexing cookies.sqlite
Reindexing downloads.sqlite
Reindexing formhistory.sqlite
Reindexing permissions.sqlite
Reindexing places.sqlite
Reindexing search.sqlite
Reindexing signons.sqlite
Reindexing urlclassifier3.sqlite
chris@ubuntu:~/.mozilla/firefox/1u64q3v3.default$