B.1: Introduction

• Main memory is usually made from dynamic ram (DRAM) which has a typical access time of 30 ns or more.
• In the 5-stage pipeline we assume that an instruction fetch or a memory read can be done in one cycle.
• One cycle is typically a fraction of a nanosecond.
• It does not make sense to stall for 50 to 100 cycles each time we do an instruction fetch.
• This problem is solved by using a cache, a relatively small fast memory that holds a subset of the contents of the main memory.
• Assume we have a cache that can be accessed in one cycle.
  When an access is attempted:
  • first look in the cache
  • if there, we are done. This is called a cache hit.
  • if not, we look in main memory. This is called a cache miss.
• The number of extra cycles needed during a cache miss is called the miss penalty.
• The fraction of time that an access is found in the cache is called the cache hit ratio.
• The fraction of time that an access is not found in the cache is called the cache miss ratio.

Memory Hierarchy

• The fastest memory of a computer system is the registers.
• Compilers try to keep as much data in registers to avoid memory accesses.
• If memory accesses cannot be satisfied in the cache, main memory must be accessed.
• If there is not enough main memory, some data will need to be stored on disk.
• Modern computers have several levels of cache between the registers and main memory.

• Main memory and disks are organized so that after a latency, a block can be transferred quickly.
• When a cache miss occurs, a block is transfered to the cache.
• Temporal locality: the same data is likely to be needed again soon.
• Spatial locality: data near recently accessed data will be needed soon.
• The miss penalty depends on both the latency and the bandwidth.
• With in-order execution, a cache miss causes the pipeline to stall.
• With out-of-order execution, a stall of subsequent instructions might not be necessary.

A direct mapped cache example

• Suppose we have a cache block size of 64 bytes and a cache with 256 blocks.
• The size of the cache is 64 × 256 = 16K.
• A memory address looks like this:

       ---------------------------------
      |       tag    |  index  | offset |
       ---------------------------------
                       8 bits   6 bits
  

• The offset is the offset into the cache block.
• The index determines where in the cache the block will be stored.
• If a memory address is 32 bits, the tag is 32 - 8 - 6 bits = 18 bits.
• A cache block looks like this:

       ---------------------------------
      |       tag    |     data         |
       ---------------------------------
          18 bits         64 bytes
  

• We need to store the tag with the data since several different memory addresses will correspond to the same cache entry.
• We also need one valid bit to indicate that the entry contains valid data (not shown).
• If the memory address is 0x23456, what is the index and what is the tag?
  • We need to write the address in binary: 0000 0000 0000 0010 0011 0100 0101 0110
  • We can rewrite this as: 000000000000001000 11010001 010110
  • The low 6 bits are the offset: 010110 = 0x16.
  • The next 8 bits is the index: 11010001 = 0xd1
  • The rest is the tag: 000000000000001000 = 00 0000 0000 0000 0100 = 0x8
• To look up the byte with address 0x23456 in the cache:
  • find the entry at index 0xd1
  • if the valid bit is clear, the data is not in the cache.
  • otherwise, compare the tag in the entry to the tag of this address
  • if they match, the data is in the cache
• The byte at the address 0x33456 will be stored in the same cache location.

Four Questions on Cache Design

1. block placement: Where can a block be stored in the cache?
2. block identification: How is a block located in the cache?
3. block replacement: What block should be replaced on a miss?
4. write strategy: What happens on a write?

Block Placement

• The simplest method is the direct mapped cache:
  • Given an address it can only be in one position in the cache.
  • The mapping is given by:
    block position = (Block Address) MOD (Number of blocks in the cache)
  • Note that the Block Address is the address with the offset bits removed.
  • Note that the block position is just the index.
• If a block can be placed anywhere in the cache, the cache is called fully associative
• If a block is be restricted to a set of places in the cache, the cache is called set associative
  • A set is a group of blocks in the cache.
  • If there are n blocks in a set, the cache is called n-way set associative
  • The larger the value of n, the more hardware is necessary to do a fast lookup.
  • A block is mapped into a set, but then can be placed anywhere in the set.
  • Usual mapping is called bit selection:
    block position = (Block Address) MOD (Number of sets in the cache)
• A fully associative cache with m blocks is m-way set associative.
• Most real caches are direct mapped, 2-way set associative, or 4-way set associative.

Block identification

Write strategy

• No change to the cache or memory is necessary during a read hit.
• Reads are the common case in cache accesses.
• Each instruction requires a read during the instruction fetch.
• Usually separate caches are used for instruction and data access (loads and stores).
• Even among data accesses, loads are typically more than twice as common as stores.

• write-through: information is written both to the cache and the (lower-level) memory.
  Easier to implement than write-back, especially for multiprocessors.
  Can cause a write stall while waiting for write to memory to complete.
• write-back: information is written only to the cache.
  The lower-level memory is only updated during replacement.
  Requires use of a dirty bit.
  Requires less memory bandwidth than write-through and can use block transfers.
  A read miss might require a write to memory.

• write allocate: fill the cache with the required data, as done with a read miss.
• no-write allocate: never fill the cache on a write miss.

Suppose addresses are 40 bits, the cache is 64K and uses 64-byte blocks with 2-way set associative placement.
What is the format of a memory address? That is, which bits are used for the tag, index, block address, and offset?
Also, what is the maximum number of gigabytes this processor can access?
Solution:

The address is 40 bits. 64 = 2⁶ so the block offset is 6 bits.
The index indicates which set to use. There are 64K/64 = 1K blocks with 2 blocks per set, or 512 = 2⁹ sets. The index is 9 bits.
The remainder: 40 - 9 - 6 = 25 bits are for the tag.
The block address is the tag and the index.
The maximum number of gigabytes that can be accessed is 2⁴⁰/2³⁰ = 1K gigabytes, (or twice this).

The Opteron data cache