It’s not easy to find some sort of public documentation or paper that shows how a texture cache work in a GPU, most of the stuff I read so far is covered by some nda.
As fair as I understand this work:
http://portal.acm.org/citation.cfm?id=264152&dl=GUIDE&dl=ACM
describes a cache architecture which is kind of similar to what you can find in modern GPUs.

Marco

I am looking for good material about texture caching on GPU. I have some doubts about how it is actually done, if by lines/columns or regions. Can you indicate me some?

Thanks for your help,
Cristina

It’s possible to use shared memory to optimize image processing operations they way Marco describes, in fact, we have a few SDK examples showing that:

http://developer.download.nvidia.com/compute/cuda/sdk/website/samples.html

In particular, the separable convolution example comes with a white-paper that shows how to use shared memory to save bandwidth by loading elements shared by different pixels only once:

http://developer.download.nvidia.com/compute/cuda/sdk/website/projects/convolutionSeparable/doc/convolutionSeparable.pdf

Marco: The person I spoke to may have been one of the authors on that paper, but unfortunately I don’t remember his name

As I mentioned, it’s easy to construct/observe cases (even simple convolution) where its possible to use software-managed local memories to get speed increases (usually only on the order of 1-3 times in my experience, but that can be significant). Conversely however it’s also easy to construct cases that perform poorly on Cell in particular. For example, programs with random-access reads of global memory – particularly within tight control flow – perform poorly compared to GPUs, even with a software texture cache in local memory. Thus things like (fragment) shading on the SPUs is actually fairly inefficient, although certainly GPUs have been optimized for this case

It’s true that box filters used recursively can do many cool things (approaching gaussian kernels), but actually the “specialness” of that case has one other implication: there are better ways to evaluate medium-to-large box filters than “brute force”. I’d argue that the “equivalent” way (to your optimized local memories example) to evaluate a box filter on GPUs for instance is to build a summed-area table (separable parallel scan -> O(N) for NxN elements) and then use it to evaluate arbitrarily large filters (O(1) per element). Indeed this implementation will probably surpass the speed of even the optimized “local memories” version for sufficiently large filters, and boasts excellent memory coherence for the case of constant-sized filters.

That said, the most efficient way to implement “scan” on GPUs right now (at least in CUDA) involves using the local memories

Anyways I’m still a fan of software-managed caches (particularly at RapidMind we often *know* what data is coming up next, and thus can use – for example – Cell hardware very efficiently while we just have to trust GPUs to “do the right thing), but it’s not clear to me that they are necessarily superior on a theoretical level. They may be, but I need more convincing

The other orthogonal question is one of programming models. As I alluded to in the previous reply, there’s no reason why local memories have to be exposed as they are on Cell. Indeed there are nicer programming abstractions that can very effectively scale to N-level memory hierarchies without any application-programmer intervention. It may yet be another case where a higher-level abstraction will eventually lead to more efficient code, once the compilers are sufficiently mature.

Anyways, interesting stuff Your blog so far has a 100% hit ratio on extremely relevant and thought-provoking material… keep it up!

Moreover I have to agree with Uttar here, on G80 (and even more on R600..) your bottleneck is likely to be in the texture caches, while on CELL you can load a full vec4 per clock cycle and do some work on it knowing that no other event in the system can stop you from doing what you’re doing. With G80 and its local-shared memory things are much more complicated if you don’t want to find your ALUs stepping over its each other feet all the time. (yeah, ALUs do have feet )
Anyway given the company you work for I guess you know this stuff much better than me

I understand your point of view, but as a console programmer I’m not scared of devoting a lot of time to fine tune my algorithms for a specific game and platform. At the same time I can appreciate that this model doesn’t work very well in many other industries.. thus the need for having solutions that might be less efficient but much more effective when real world constraints are applied. (That’s why I think Intel is on something with Larrabee and its programming model..)

to lycium: you are right, what I wrote only applies to box filtering algorithms, but I was not obviously trying to imply it works with everything. Still current GPUs can’t implement anything like that efficiently in one rendering pass, and also we should remember that box filters used recursively let us do a lot of nice things

With scratchpads like on G80 and CELL, you can retrieve a lot of data per cycle to your ALUs. It’s practically as fast as the register file on G80… On the other hand, for every point sampled RGBA8 request on the 8800 GTX, you have the time to do 9.4 scalar operations (excluding the MUL). For scalar FP32, this is doubled, for reasons I will not go into here.

This is a topic that came up with Marco when I was discussing his shadowing algorithms with him. Even if he had 100% cache hits, my conclusion was that point sampling would be the bottleneck, not the ALUs.

I just looked at the CUDA convolution example, and indeed, this confirms my suspicion:
#if 0
// try this to see the benefit of using shared memory
int pixel = getPixel(g_data, x+dx, y+dy, imgw, imgh);
#else
int pixel = SMEM(r+tx+dx, r+ty+dy);
#endif

// only sum pixels within disc-shaped kernel
float l = dx*dx + dy*dy;
if (l >8)&0xff);
float b = float((pixel>>16)&0xff);
#if 1
// brighten highlights
float lum = (r + g + b) / (255*3);
if (lum > threshold) {
r *= highlight;
g *= highlight;
b *= highlight;
}
#endif
rsum += r;
gsum += g;
bsum += b;
samples += 1.0;
}