efficient linear algebra for

The very first non-trivial patch I ever produced for Sage was about interfacing with NTL for dense linear algebra over (I was interested in algebraic attacks against AES at the time). Here’s William’s reply:

Your NTL patch worked perfectly for me first try. I tried more benchmarks (on Pentium-M 1.8Ghz).

[...]

This is pretty good; vastly better than what’s was in SAGE by default, and way better than PARI. Note that MAGMA is much faster though (nearly 8 times faster):

[...]

MAGMA uses (1) [...] and (2) a totally different algorithm for computing the echelon form. [...] As far as I know, the MAGMA method is not implemented anywhere in the open source world But I’d love to be wrong about that… or even remedy that.

Well, that was 2006. Fast forward to the year 2011 and we get the following timings for computing the reduced row echelon form of a 1,000 x 1,000 matrix over : Sage 4.7.2 takes 36.53 seconds , NTL 5.4.2 takes 31.06 seconds and Magma 2.15 does it in 0.87 seconds. So essentially, the situation didn’t change at all for the better.

With Sage 4.8 this situation changes dramatically  and we get that Sage performs this computation in 0.08 seconds, that’s 450 times faster than Sage 4.7.2. This is because M4RIE was merged in Sage 4.8. Hence, Sage is now (in some cases by far) fastest system to do linear algebra with dense matrices over for  and usually also for .

efficient linear algebra for

One can tell a similar story for for, say, small to medium sized primes . In Sage 4.7.2 it took 1.12 seconds to multiply two 1,000 x 1,000 matrices over (although you always had the option to call LinBox explicitly which was way faster but took more memory). With Sage 4.8 the same computation takes 0.16 seconds. For comparison, Magma 2.15 takes 0.22 seconds. So here again Sage moved from poor performance to best in class performance between 4.7.2 and 4.8 simply by making proper use of available libraries.

Viable Alternative yet?

Overall, the story for dense linear algebra in Sage for small finite fields  is as follows.

	Implementation	Comments
	M4RI	Fastest implementation or equal performance depending on platform
	LinBox	Decent performance, but faster implementations are known in the literature. Also, Magma is a bit faster on my machine.
prime <	LinBox	Fastest implementation or equal performance depending on platform.
for	M4RIE	Fastest
for	Generic	Very poor performance, but some work is being done.

So, once we fix that last row Sage finally achieves “viable alternative” quality when it comes to dense linear algebra over if  is .

Abstract: Transforming a matrix over a field to echelon form, or decomposing the matrix as a product of structured matrices that reveal the rank profile, is a fundamental building block of computational exact linear algebra. This paper surveys the well known variations of such decompositions and transformations that have been proposed in the literature. We present an algorithm to compute the CUP decomposition of a matrix, adapted from the LSP algorithm of Ibarra, Moran and Hui (1982), and show reductions from the other most common Gaussian elimination based matrix transformations and decompositions to the CUP decomposition. We discuss the advantages of the CUP algorithm over other existing algorithms by studying time and space complexities: the asymptotic time complexity is rank sensitive, and comparing the constants of the leading terms, the algorithms for computing matrix invariants based on the CUP decomposition are always at least as good except in one case. We also show that the CUP algorithm, as well as the computation of other invariants such as transformation to reduced column echelon form using the CUP algorithm, all work in place, allowing for example to compute the inverse of a matrix on the same storage as the input matrix.

http://arxiv.org/abs/1112.5717

In terms of coding projects, first, I tried to speed up linear algebra mod p where p is a 32 or 64 bit prime. But it turns out that any trick I could think of could not improve on Frederik’s code. So that didn’t lead anywhere but I allowed me to read some code of FLINT2 (very readable) and admire how carefully it is written.

My other two projects both involved evaluate–pointwise-multiply–interpolate algorithms for fast matrix-matrix products over finite extension fields or for matrices with polynomial coefficients (over prime fields).  After my talk on M4RI(E) David Harvey worked out how to improve multiplication over from 17 multiplications over to 15, which then lead to a general approach for with composite . Much of it remains to be implemented (efficiently), but the example indeed shows a 10% speed-up as expected. The code is not clean yet, uses way too much memory and doesn’t deal with the more advanced finite field stuff appropriately. It should end up in M4RIE eventually though.

I also contributed a bit to #12177 which is about a “prime slice” implementation of matrices over . The idea is essentially to represent  these matrices as polynomials with matrix coefficients and to use fast polynomial multiplication algorithms for these polynomials. It turns out, this works very well even for small finite fields. Burcin Eröcal did all the coding, I only helped with some discussions. We need to polish the code a lot to be usable, so if you like matrices over head over to #12177 and help out.

https://bitbucket.org/malb/algebraic_attacks/src/1af75effcc7d/coldboot

For this code to run you’ll need to apply this patch to Sage:

http://trac.sagemath.org/sage_trac/ticket/10879

which adds an interface to SCIP. Unfortunately, this patch crashes on OSX and I didn’t figure out yet why. Anybody willing to help, please step forward

Also, I assume the code on bitbucket needs some patching to work with the most recent version of Sage. Patches very welcome!

Abstract: In this work, we present the M4RIE library which implements efficient algorithms for linear algebra with dense matrices over for . As the name of the library indicates, it makes heavy use of the M4RI library both directly (i.e., by calling it) and indirectly (i.e., by using its concepts). We provide an open-source GPLv2+ C library for efficient linear algebra over for e small. In this library we implemented an idea due to Bradshaw and Boothby which reduces matrix multiplication over to a series of matrix multiplications over . Furthermore, we propose a caching technique – Newton-John tables – to avoid finite field multiplications which is inspired by Kronrod’s method (“M4RM”) for matrix multiplication over . Using these two techniques we provide asymptotically fast triangular solving with matrices (TRSM) and PLE-based Gaussian elimination. As a result, we are able to significantly improve upon the state of the art in dense linear algebra over with .

Abstract: In this work we describe an efficient implementation of a hierarchy of algorithms for Gaussian elimination upon dense matrices over the field with two elements (). We discuss both well-known and new algorithms as well as our implementations in the M4RI library, which has been adopted into Sage. The focus of our discussion is a block iterative algorithm for PLE decomposition which is inspired by the M4RI algorithm. The implementation presented in this work provides considerable performance gains in practice when compared to the previously fastest implementation. We provide performance figures on x86_64 CPUs to demonstrate the alacrity of our approach.

The sources of this document are available on bitbucket. But I also compiled a PDF.

Update: arXiv link.

We did not discuss the paper  in great detail, but Jake did mention one interesting avenue for continued research. Given that this new approach allows one to cast both LWE and approximate GCD in the same framework, can one also capture ring-LWE. If so, this might enable a better comparison of the various fully homomorphic encryption (FHE) schemes out there. The hope expressed by Jake was that this might allow a reduction to standard LWE (for the current batch of ring-LWE schemes), which would boost our confidence in those schemes.

This motivated me to express the Ring-LWE problem in a language of Gröbner bases, here’s what I could come up with so far.

But let’s recall the computational Ring-LWE problem first: Given a prime and an ideal where is a power of two and , we consider the quotient ring . We pick a random element which is our secret. We sample tuples where are random and are sampled according to some noise distribution. The computational Ring-LWE problem is to compute given .

Assume for all , so that we can discuss the algebraic structure directly. Clearly, all are in the ideal spanned by in . Furthermore, there is a direct correspondence between ideals in the quotient ring and in . Hence, to recover the Gröbner basis for the ideal spanned by , we simply compute the Gröbner basis of , easy right? Except that the Gröbner basis will be … wait for it … 1 with very high probability. This might reduce the search space slightly (since it tells us that and have no common factors over ) and is correct (since one is the smallest representative of the ideal spanned by in ) this is not terribly useful. But we did ignore so far.

Namely, the problem actually is to compute or . Now, in order to compute in – which is defined if – we may run an extended GCD algorithm which returns for inputs such that . Hence, for our inputs it will compute and thus .

In the language of Gröbner bases the extended GCD equivalent is often called “lifting”: Given an ideal and some , find such that .  The problem is easy given a Gröbner basis (in our case ), since every element can be written as where .  In general, it might be hard because the a priori bound on the degree of the output may be large. In any case, instead of solving a(n approximate) GCD (or GB(N)), we are now solving an extended GCD (or lifting with GB(N)), i.e., we keep track of our computation. Well, here’s an example in Sage:

sage: n = 2^3
sage: q = 17
sage: R. = GF(q)[]
sage: Q. = R.quotient(Xbar^n + 1)
sage: s = Q.random_element()
sage: s
8*X^7 + 4*X^6 + 11*X^5 + 6*X^4 + 15*X^3 + 12*X^2 + 14*X + 12
sage: a = Q.random_element()
sage: P. = PolynomialRing(GF(q),1)
sage: A = sage_eval(str(a),{'X':x})
sage: S = sage_eval(str(s),{'X':x})
sage: Ainv = P(1).lift( (A,x^n + 1) )[0]
sage: Ainv*(A*S) % (x^n + 1)
8*x^7 + 4*x^6 - 6*x^5 + 6*x^4 - 2*x^3 - 5*x^2 - 3*x - 5

Alternatively, we may “lift” with respect to directly. Well, I don’t know if any of the above is actually useful, as I said that’s how far I got after reading Martijn’s blog post.

See http://wiki.sagemath.org/SageFlintDays for more information and registration.

PS: I’ll be talking about M4RI(E) … big surprise.

The first three rows are from GB computations for the hidden field equations cryptosystem (those matrices were provided by Michael Brickenstein). The “mutant” row is a matrix as it appears during a run of the MXL2 algorithm on a random system (I believe). It was contributed by Wael Said. The rows “random n=25,m=26″ are matrices as they appear during a GB computation with PolyBoRi for a random system of equations in 24 variables and 26 equations. The remaining rows are matrices from PolyBoRi computations on small scale AES instances. Those rows which have “compressed” in their description correspond to systems where “linear variables” were eliminate before running the Gröbner basis algorithm.

The last three columns give running times (quite rough ones!) for computing an echelon form (not reduced) using (a) the M4RI algorithm, (b) PLE decomposition and (c) a first implementation of the TRSM for trivial pivots trick. As you can see, currently it’s not straight-forward to pick which strategy to use to eliminate matrices appearing during Gröbner basis computations: the best algorithm to pick varies between different problems and the differences can be dramatic.