On a steep rock in the middle of the town of Kamnik stand the remains of the Little Castle. A part of the wall is preserved and kept up, and a stout tower with a four-sided ground plan and a two-storey chapel with a tower have been built anew (1993). Under the stone walls are the remains of a Slavic burial ground. The castle was mentioned in written sources in 1202. It was run by the Kamnik knights. The fortification was torn down in the 16th century, but the chapel is preserved. The building, along with a crypt, was probably built around 1100 - the same as the Chapel of St. Margaret (present presbytery of the lower chapel). Soon a nave was added and the Romanesque portal with a lunette was moved.  The portal ends in a semicircle and in the lunette it is decorated with symbolic relief. The chapel was added on in the 13th century. The upper chapel was dedicated to St Bartholomew (sv. Jernej) and the lower to St. Elijah (sv. Eligij). The new staircase to the chapel surmounted the archaeological finds and the reconstructed walls.

The upper chapel was vaulted in Gothic style after 1470. The entire thing was made into the Baroque style around 1700. The lower chapel in the presbytery was decorated with frescos after 1771 (painter Janez Potočnik). Fragments of older frescos in the nave depict images of St. Erasmus (sv. Erazem) and donators. The painting originated around 1520 in the circle of Master Vid. In the presbytery there are still remains of frescos from the 15th century.

The platform of the castle is open regularly; the interior is accessible only by prior arrangement. The chapel is open from june until september.