summaryrefslogtreecommitdiffstats
path: root/drivers/iommu/io-pgtable-arm-v7s.c (follow)
Commit message (Collapse)AuthorAgeFilesLines
* iommu/io-pgtable: Sanitise map/unmap addressesRobin Murphy2017-07-201-0/+6
| | | | | | | | | | | | | It may be an egregious error to attempt to use addresses outside the range of the pagetable format, but that still doesn't mean we should merrily wreak havoc by silently mapping/unmapping whatever truncated portions of them might happen to correspond to real addresses. Add some up-front checks to sanitise our inputs so that buggy callers don't invite potential memory corruption. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm: Use dma_wmb() instead of wmb() when publishing tableWill Deacon2017-06-231-2/+6
| | | | | | | | | | | | | | | | | | | When writing a new table entry, we must ensure that the contents of the table is made visible to the SMMU page table walker before the updated table entry itself. This is currently achieved using wmb(), which expands to an expensive and unnecessary DSB instruction. Ideally, we'd just use cmpxchg64_release when writing the table entry, but this doesn't have memory ordering semantics on !SMP systems. Instead, use dma_wmb(), which emits DMB OSHST. Strictly speaking, this does more than we require (since it targets the outer-shareable domain), but it's likely to be significantly faster than the DSB approach. Reported-by: Linu Cherian <linu.cherian@cavium.com> Suggested-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: Support lockless operationRobin Murphy2017-06-231-21/+63
| | | | | | | | | | | | | | | | | | Mirroring the LPAE implementation, rework the v7s code to be robust against concurrent operations. The same two potential races exist, and are solved in the same manner, with the fixed 2-level structure making life ever so slightly simpler. What complicates matters compared to LPAE, however, is large page entries, since we can't update a block of 16 PTEs atomically, nor assume available software bits to do clever things with. As most users are never likely to do partial unmaps anyway (due to DMA API rules), it doesn't seem unreasonable for this case to remain behind a serialising lock; we just pull said lock down into the bowels of the implementation so it's well out of the way of the normal call paths. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable: Introduce explicit coherencyRobin Murphy2017-06-231-7/+10
| | | | | | | | | | | | | | | | | | | | | Once we remove the serialising spinlock, a potential race opens up for non-coherent IOMMUs whereby a caller of .map() can be sure that cache maintenance has been performed on their new PTE, but will have no guarantee that such maintenance for table entries above it has actually completed (e.g. if another CPU took an interrupt immediately after writing the table entry, but before initiating the DMA sync). Handling this race safely will add some potentially non-trivial overhead to installing a table entry, which we would much rather avoid on coherent systems where it will be unnecessary, and where we are stirivng to minimise latency by removing the locking in the first place. To that end, let's introduce an explicit notion of cache-coherency to io-pgtable, such that we will be able to avoid penalising IOMMUs which know enough to know when they are coherent. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: Refactor split_blk_unmapRobin Murphy2017-06-231-40/+45
| | | | | | | | | | | Whilst the short-descriptor format's split_blk_unmap implementation has no need to be recursive, it followed the pattern of the LPAE version anyway for the sake of consistency. With the latter now reworked for both efficiency and future scalability improvements, tweak the former similarly, not least to make it less obtuse. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: Check table PTEs more preciselyRobin Murphy2017-06-231-1/+2
| | | | | | | | | Whilst we don't support the PXN bit at all, so should never encounter a level 1 section or supersection PTE with it set, it would still be wise to check both table type bits to resolve any theoretical ambiguity. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: constify dummy_tlb_ops.Arvind Yadav2017-06-231-1/+1
| | | | | | | | | | | | | File size before: text data bss dec hex filename 6146 56 9 6211 1843 drivers/iommu/io-pgtable-arm-v7s.o File size After adding 'const': text data bss dec hex filename 6170 24 9 6203 183b drivers/iommu/io-pgtable-arm-v7s.o Signed-off-by: Arvind Yadav <arvind.yadav.cs@gmail.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: Check for leaf entry before dereferencing itOleksandr Tyshchenko2017-03-101-1/+5
| | | | | | | | | | | Do a check for already installed leaf entry at the current level before dereferencing it in order to avoid walking the page table down with wrong pointer to the next level. Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> CC: Will Deacon <will.deacon@arm.com> CC: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: Add support for the IOMMU_PRIV flagRobin Murphy2017-01-191-1/+5
| | | | | | | | | The short-descriptor format also allows privileged-only mappings, so let's wire it up. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Tested-by: Sricharan R <sricharan@codeaurora.org> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm: Use for_each_set_bit to simplify the codeKefeng Wang2016-11-291-4/+1
| | | | | | | | | We can use for_each_set_bit() to simplify the code slightly in the ARM io-pgtable self tests. Reviewed-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm: Check for v7s-incapable systemsRobin Murphy2016-09-161-0/+4
| | | | | | | | | | | On machines with no 32-bit addressable RAM whatsoever, we shouldn't even touch the v7s format as it's never going to work. Fixes: e5fc9753b1a8 ("iommu/io-pgtable: Add ARMv7 short descriptor support") Reported-by: Eric Auger <eric.auger@redhat.com> Tested-by: Eric Auger <eric.auger@redhat.com> Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: Fix attributes when splitting blocksRobin Murphy2016-08-191-1/+3
| | | | | | | | | | | | | | | | | | | | Due to the attribute bits being all over the place in the different types of short-descriptor PTEs, when remapping an existing entry, e.g. splitting a section into pages, we take the approach of decomposing the PTE attributes back to the IOMMU API flags to start from scratch. On inspection, though, the existing code seems to have got the read-only bit backwards and ignored the XN bit. How embarrassing... Fortunately the primary user so far, the Mediatek IOMMU, both never splits blocks (because it only serves non-overlapping DMA API calls) and also ignores permissions anyway, but let's put things right before any future users trip up. Cc: <stable@vger.kernel.org> Fixes: e5fc9753b1a8 ("iommu/io-pgtable: Add ARMv7 short descriptor support") Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable-arm-v7s: Support IOMMU_MMIO flagRobin Murphy2016-04-071-5/+11
| | | | | | | Teach the short-descriptor format to create Device mappings when asked. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* iommu/io-pgtable: Add MTK 4GB mode in Short-descriptorYong Wu2016-04-051-1/+12
| | | | | | | | | | | | | | | | | | | | | | | | In MT8173, Normally the first 1GB PA is for the HW SRAM and Regs, so the PA will be 33bits if the dram size is 4GB. We have a "DRAM 4GB mode" toggle bit for this. If it's enabled, from CPU's point of view, the dram PA will be from 0x1_00000000~0x1_ffffffff. In short descriptor, the pagetable descriptor is always 32bit. Mediatek extend bit9 in the lvl1 and lvl2 pgtable descriptor as the 4GB mode. In the 4GB mode, the bit9 must be set, then M4U help add 0x1_00000000 based on the PA in pagetable. Thus the M4U output address to EMI is always 33bits(the input address is still 32bits). We add a special quirk for this MTK-4GB mode. And in the standard spec, Bit9 in the lvl1 is "IMPLEMENTATION DEFINED", while it's AP[2] in the lvl2, therefore if this quirk is enabled, NO_PERMS is also expected. Signed-off-by: Yong Wu <yong.wu@mediatek.com> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* iommu/io-pgtable-armv7s: Fix kmem_cache_alloc() flagsRobin Murphy2016-03-021-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Whilst the default SLUB allocator happily just merges the original allocation flags from kmem_cache_create() with those passed through kmem_cache_alloc(), there is a code path in the SLAB allocator which will aggressively BUG_ON() if the cache was created with SLAB_CACHE_DMA but GFP_DMA is not specified for an allocation: kernel BUG at mm/slab.c:2536! Internal error: Oops - BUG: 0 [#1] SMP ARM Modules linked in:[ 1.299311] Modules linked in: CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.5.0-rc6-koelsch-05892-ge7e45ad53ab6795e #2270 Hardware name: Generic R8A7791 (Flattened Device Tree) task: ef422040 ti: ef442000 task.ti: ef442000 PC is at cache_alloc_refill+0x2a0/0x530 LR is at _raw_spin_unlock+0x8/0xc ... [<c02c6928>] (cache_alloc_refill) from [<c02c6630>] (kmem_cache_alloc+0x7c/0xd4) [<c02c6630>] (kmem_cache_alloc) from [<c04444bc>] (__arm_v7s_alloc_table+0x5c/0x278) [<c04444bc>] (__arm_v7s_alloc_table) from [<c0444e1c>] (__arm_v7s_map.constprop.6+0x68/0x25c) [<c0444e1c>] (__arm_v7s_map.constprop.6) from [<c0445044>] (arm_v7s_map+0x34/0xa4) [<c0445044>] (arm_v7s_map) from [<c0c18ee4>] (arm_v7s_do_selftests+0x140/0x418) [<c0c18ee4>] (arm_v7s_do_selftests) from [<c0201760>] (do_one_initcall+0x100/0x1b4) [<c0201760>] (do_one_initcall) from [<c0c00d4c>] (kernel_init_freeable+0x120/0x1e8) [<c0c00d4c>] (kernel_init_freeable) from [<c067a364>] (kernel_init+0x8/0xec) [<c067a364>] (kernel_init) from [<c0206b68>] (ret_from_fork+0x14/0x2c) Code: 1a000003 e7f001f2 e3130001 0a000000 (e7f001f2) ---[ end trace 190f6f6b84352efd ]--- Keep the peace by adding GFP_DMA when allocating a table. Reported-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Robin Murphy <robin.murphy@arm.com> Tested-by: Geert Uytterhoeven <geert+renesas@glider.be> Signed-off-by: Joerg Roedel <jroedel@suse.de>
* iommu/io-pgtable: Rationalise quirk handlingRobin Murphy2016-02-171-0/+5
| | | | | | | | | | | | | | | As the number of io-pgtable implementations grows beyond 1, it's time to rationalise the quirks mechanism before things have a chance to start getting really ugly and out-of-hand. To that end: - Indicate exactly which quirks each format can/does support. - Fail creating a table if a caller wants unsupported quirks. - Properly document where each quirk applies and why. Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable: Add helper functions for TLB opsRobin Murphy2016-02-171-28/+20
| | | | | | | | | | | Add some simple wrappers to avoid having the guts of the TLB operations spilled all over the page table implementations, and to provide a point to implement extra common functionality. Acked-by: Will Deacon <will.deacon@arm.com> Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com> Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
* iommu/io-pgtable: Add ARMv7 short descriptor supportRobin Murphy2016-02-171-0/+849
Add a nearly-complete ARMv7 short descriptor implementation, omitting only a few legacy and CPU-centric aspects which shouldn't be necessary for IOMMU API use anyway. Reviewed-by: Yong Wu <yong.wu@mediatek.com> Tested-by: Yong Wu <yong.wu@mediatek.com> Signed-off-by: Yong Wu <yong.wu@mediatek.com> Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com>