In PS you would need to first run the Find Edges filter, then desaturate it. If you are going to use it in PS to mask a sharpening filter then you copy the original layer and then just working in the alpha channel paste it in, it will be monochrome already when you do this, so just run Find edges.
With the edges image you then need to run a Levels command (Ctrl L). You will need to bring both ends inwards to make the lines more defined. I usually use 192/.10/230 as my starting point when making an edge filter for a highpass sharpening layer. I also usually keep everything as black on white until the end of the process, then invert to black in white at the end using Ctrl I.
If you look in the Sharpening sticky above there is a really good tutorial on this, with lots of example images. Although aimed at edge sharpening I also like the effect you get from doing this in colour as a layer over the original using the Soft light blend mode. You get a kind of watercolour effect but with a defined almost drawn outline. If you do that as well as keeping it in colour, don't go nearly as far with the sliders in the Levels operation, and don't invert at the end.
Alan